---
output:
  pdf_document:
    fig_caption: yes
    toc: true
    toc_depth: 3
title: "Online Appendix"
author: 
 - Joshua Kalla^[Yale University, Departments of Political Science and Statistics & Data Science, josh.kalla@yale.edu]
 - David Broockman^[University of California, Berkeley, Department of Political Science, dbroockman@berkeley.edu]
#geometry: margin=1in
#fontfamily: mathpazo
#fontsize: 12pt
#header-includes:
#    - \linespread{2}
#    - \renewcommand{\footnotesize}{\normalsize}
header-includes:
  - \usepackage{pdfpages}
  - \usepackage{caption} 
  - \usepackage{float}
  - \usepackage{comment}
---

\setcounter{page}{1}

\setcounter{table}{0}
\renewcommand\thetable{OA\arabic{table}}

\setcounter{figure}{0}
\renewcommand\thefigure{OA\arabic{figure}}

```{r setup, include=FALSE}
knitr::opts_chunk$set(echo = TRUE)
```

```{r load, echo=FALSE, message=FALSE, warning=FALSE, include = FALSE}
#Load Libraries
library(knitr)
library(pander)
library(bookdown)
library(captioner)
library(sandwich)
library(lmtest)
library(kableExtra)
library(stringr)
library(ggplot2)
library(gridExtra)
library(grid)
library(tidyverse)
library(plyr)
library(ebal)
library(scales)

setwd("")
data <- read.csv("tv_experiment_sm_data.csv", stringsAsFactors = FALSE)
```

Pre-analysis plan and TV ad scripts are available at https://osf.io/gw8b7/?view_only=6c206c3d79694d93a9bf6322eb3eab4b.

Replication data is available at https://doi.org/10.7910/DVN/VELEMB.

# Ethical Considerations

The authors declare that the human subjects research in this article was reviewed by the Yale University Human Subjects Committee. The authors affirm that this article adheres to the APSA's Principles and Guidance on Human Subjects Research. Participants provided informed consent and were compensated $5 for their participation.

# Description of the Experimental Setting

Our analysis focuses on a television advertising campaign conducted by national LGBTQ and immigrant rights groups in Spring 2020.^[This experiment was conducted early in the COVID-19 pandemic, a period of increased television viewership (https://perma.cc/EVY3-KUYS).] These advertisements intended to educate the public about the issue of immigrant and transgender rights, increase support for these rights, and secondarily, to increase the likelihood that individuals would take political action (e.g., contacting an elected official) to advocate for these rights. The issues, advertisements, airing dates, airing volumes, states, and targeted households were all determined by the LGBTQ and immigrant rights groups. The partner organizations provided all funding for the surveys and advertising. Permission to publish was received at the start of the project, prior to results, in order to minimize the risk of publication bias. 

While the advertisements ran, we are unaware of any other large television advertising campaigns about LGBTQ or immigration issues (either for or against). For example, 95% of respondents in the baseline survey did not recall seeing ads on television recently about "how businesses treat gay and lesbian people." Similarly, 85% of respondents in the baseline survey did not recall seeing ads on television recently about "how the United States treats unauthorized immigrants and asylum seekers." Other than the advertising from this experiment, it appears unlikely that many respondents were seeing other, unrelated advertising about immigration or LGBTQ rights. This is consistent of typical issue advertising campaigns. As Falk, Grizard and McDonald (2005) find, most issue advertising campaigns are one-sided. Rarely does an issue attract advertising both supporting and opposing it. 

However, many respondents were seeing advertisements about the 2020 presidential election. In the post-treatment survey, 82% recalled seeing ads on television recently about "the 2020 presidential election" (we did not ask about this in the baseline survey). It is possible that respondents were seeing presidential advertising about immigration because 22% of Trump's campaign ads mentioned the issue (Ridout et al. 2021, Table 4). LGBTQ rights was not a major theme in the advertisements of either the Trump or the Biden campaigns. However, it is unlikely that many survey respondents were seeing Trump's immigration ads. First, Ridout et al. (2021, Figure 3) shows that Trump advertising did not become common until mid-July, several months after the experiment concluded. Second, a large percentage of participants in the experiment were outside of 2020 battleground states. 11% lived in California, 14% lived in Tennessee, and 18% lived in Colorado. These individuals were unlikely to receive much presidential advertising. 

Overall, it therefore appears that the experiment was conducted in a political environment where there was a vacuum of opposing ads. Most likely the ads included in the experiment were the only supporting ads respondents were exposed to. The context is thus similar to that in Gerber et al. (2011), except about issues rather than a candidate. Gerber et al. (2011, p. 138) summarize their setting thus: "the manner in which ads were deployed closely approximates what Zaller (1996) describes as the ideal conditions for detecting media effects: well-measured, abrupt shifts in the quantity of advertising; a vacuum or profusion of opposing ads; a single ad that is deployed through the three-week experimental period, with no ads preceding or following it; and continuous tracking of opinion before, during, and after the flight of advertising. Although this experiment cannot tell us how media effects might play out under different conditions, it does speak with special clarity to the question of whether paid advertising is capable of producing noticeable shifts in voter support."

# Overview of Experimental Design

\begin{table}[H]
\caption{Experimental Design} \label{tab:design}
\begin{tabular}{p{3.6cm}| p{3.6cm} | p{3.6cm} | p{3.6cm} }
\toprule
& \multicolumn{3}{c}{\textbf{Experimental Conditions}} \\
 & \textbf{Immigration}: Eddie \& Prosperous Future & \textbf{Immigration}: MariCruz/Kandy \& Prosperous Future & \textbf{LGBTQ}: Open to All \\
\midrule
Baseline Survey (2/1-2/24, 2020) & $n =10,467$  & $n =10,468$  & $n =10,469$ \\
\hline
Ads Air (3/2-3/23) & 61\% saw ad at least once; median was 13 views.  & 61\% saw ad at least once; median was 13 views. & 62\% saw ad at least once; median was 11 views.  \\
\hline
Endline Survey Randomly Assigned to Launch... &   &   &    \\
$\bullet$ 3/21 & $n =1,906$ & $n =1,876$ & $n =1,907$ \\ 
$\bullet$ 3/24 & $n =1,799$ & $n =1,748$ & $n =1,791$ \\ 
$\bullet$ 3/26 & $n =1,688$ & $n =1,628$ & $n =1,700$ \\ \bottomrule
\end{tabular}
\end{table}

# Description of Ads

The **"Prosperous Future"** immigration ad features a middle-aged white woman sharing how she used to think all immigrants should just "get in line" but then she did her research and learned that "there is no line," that the immigration system is broken. The ad ends with the woman urging immigration reform, though no particular policies are mentioned. This advertisement aimed to feature a messenger modeling opinion change, with the theory being that a white woman with no obvious pro-immigrant vested interest would be perceived as more relatable to the audience being reached. Both of the experimental conditions assigned to receive advertisements about immigration received this advertisement.

The **"Eddie"** advertisement features a first-person narrative from Eddie, a middle-aged Latino man who came to the United States through the asylum process. He shares how he holds dear many stereotypical American values: family, hard work, and freedom. Eddie shares that since coming to America, he became a business owner who gives back to his community. He then discusses how there are many people like him who are also seeking asylum today, but for whom the asylum process is broken. The ad ends with Eddie supporting immigration and asylum reform, though no particular policies are mentioned. This ad aimed to use narrative persuasion paired with moral reframing through American values to increase support for immigrants and asylum seekers.

The **"MariCruz/Kandy"** advertisement features two co-workers, Kandy, a middle-aged white woman, and MariCruz, a middle-aged Latina woman. MariCruz shares how she came to the United States as an undocumented immigrant to provide a better future for her son. MariCruz notes that she works and pays taxes. Kandy serves as a validator for MariCruz by noting that she is a hard worker. The ad ends with Kandy sharing that she was surprised to learn that undocumented workers pay over \$11 billion in federal taxes and \$12 billion to Social Security. These facts are also displayed as visuals on the screen. This ad uses both a more neutral messenger (like in Prosperous Future) and a personal narrative (like Eddie) while sharing a specific fact.

The **"Open to All"** LGBTQ advertisement features Howard and Pat, an older couple who present as white, heterosexual, and married. Howard and Pat describe themselves as small business owners and Christians. They note that treating people how they want to be treated is both good for business and what their Christian faith expects of them. Howard notes that he was surprised when he learned that there are people who will not hire or do business with someone who is gay or transgender. Pat says that everyone should be treated equally under the law and that nobody should be refused services because of their sexual orientation or gender identity. 

# Representativeness

The below table shows how the representativeness of those who responded to the survey differ from those mailed an invitation to participate in the survey. These data come from the voter file.

```{r eval=TRUE, echo=FALSE}
assess.representativeness <- function(df) c(mean(df$vf_female), mean(df$vf_age),
                                            mean(df$vf_afam), mean(df$vf_white), mean(df$vf_hispanic), 
                                            mean(df$vf_general18), mean(df$vf_general16),
                                            mean(df$vf_general14), mean(df$vf_general12),
                                            mean(df$vf_dem), 
                                            mean(df$vf_rep), 
                                            mean(df$state_CA),
                                            mean(df$state_CO),
                                            mean(df$state_MI),
                                            mean(df$state_NC),
                                            mean(df$state_TN),
                                            mean(df$state_WI),
                                            nrow(df))

starting.universe <- t(c("Starting", assess.representativeness(data)))
t0.sample <- t(c("Pre-Treat Resp.", assess.representativeness(subset(data, data$t0_respondent == 1))))
t1.sample <- t(c("Post-Treat Resp.", assess.representativeness(subset(data, data$t1_respondent == 1))))

representativeness <- rbind(starting.universe, t0.sample, t1.sample)
colnames(representativeness) <- c("Sample", "Female", "Age", "AfAm", "White", "Latino", "Voted 18",
                                  "Voted 16", "Voted 14", "Voter 12", "Reg Democrat",
                                  "Reg Republican", 
                                  "CA", "CO", "MI", "NC", "TN", "WI",
                                  "N")
representativeness[, 2:18] <- round(as.numeric(representativeness[, 2:18]), 2)


kable(t(representativeness), caption = "Representativeness of Experiment at Each Stage")  %>%
  kable_styling(latex_options = c("striped", "hold_position"))
```

# Outcomes

The survey included dozens of political, social, and cultural questions, only some of which were related to immigration and LGBTQ rights In our pre-analysis plan, we indicate which items constituted experimental outcomes and which outcomes were exploratory. Below we list these items and give their full text.

In our pre-analysis plan, we specified that we would combine multiple items into indices to test hypotheses. Combining outcomes into an index increases precision by decreasing survey measurement error and limits the potential for biases from multiple hypothesis testing (Broockman, Kalla, and Sekhon 2017). We formed these indices by taking the first dimension from a factor analysis in Stata. All indices and individual exploratory items are standardized to have mean 0 and standard deviation 1. We also code all indices such that positive values indicate the intended direction of the treatment effects.

In addition to the below indices, we also formed overall immigration, LGBTQ, and asylum indices with all of the items from each prejudice and policy index.

Except where otherwise noted, the questions came from matrix grids with five point scales where respondents were asked: "Do you agree or disagree with the below statements [example: about undocumented or illegal immigrants]?" Response options were: Strongly agree (coded as 5), Somewhat agree (coded as 4), Neither agree nor disagree (coded as 3), Somewhat disagree (coded as 2), Strongly disagree (coded as 1).

* Immigration Prejudice Index.
* `t1_immprej_therm`: 0-100 feeling thermometer: Someone who is an illegal immigrant.
* `t1_immprej_burdenoncomm`: Undocumented immigrants are a burden on our community.
* `t1_immprej_contributepos`: Undocumented immigrants contribute positively to our communities.
* `t1_immprej_holdsamevalues`: Undocumented immigrants hold similar values as me and my family.
* `t1_immprej_noproblemlivingnear`: I would have no problem living in areas where undocumented immigrants live.
* `t1_immprej_dontfitin`: Too many undocumented immigrants just don't want to fit into American society.
* `t1_immprej_sufferingconcernsme`: The suffering of illegal immigrants concerns me.
* `t1_immprej_deservecaresupport`: Undocumented immigrants deserve our care and support.
* Immigration Policy Index:
* `t1_immpolicy_deportall`: The U.S. government should work to identify and deport all illegal immigrants, including in the workplace.
* `t1_immpolicy_legalstatus`: The 11 million undocumented immigrants already living in the U.S. should be allowed to remain here and become citizens if they meet certain requirements over time. 
* `t1_immpolicy_donotdeserve`: Illegal immigrants have not contributed enough to deserve access to government programs.
* LGBTQ Prejudice Index:
* `t1_lgbtprej_trans_therm`: 0-100 feeling thermometer: Someone who is transgender.
* `t1_lgbtprej_gay_therm`: 0-100 feeling thermometer: Someone who is gay.
* `t1_lgbtprej_overcorrect`: Our country has over-corrected and gone too far in its attempts to give gay and transgender people equal rights.
* `t1_lgbtprej_workclose`: I would feel comfortable working closely with a gay or transgender person.
* LGBTQ Policy Index:
* `t1_lgbtpolicy_discrim`: A federal law that would protect gay and transgender people from discrimination in employment, housing, and public accommodations like stores and restaurants. 
* `t1_lgbtpolicy_fire`: A law in [STATE] protecting gay and transgender people from being fired for being gay or transgender.
* `t1_lgbtpolicy_conflict`: A law in [STATE] that would exempt business owners from following non-discrimination laws if those laws conflict with their moral or religious beliefs. 
* `t1_lgbtpolicy_refuse`: Business owners should be allowed to refuse to provide products or services to gay or transgender people if doing so violates their religious beliefs.
* `t1_lgbtpolicy_serveall`: When a business opens its doors to the public, it should serve everyone on the same terms and not discriminate. 
* `t1_lgbtpolicy_moreimpt`: A 1-6 scale asking if respondents are closer to "It is more important for the law to protect gay and transgender people from discrimination, even if that means some people may have to go against their religious beliefs to accommodate gay and transgender people" (1) or "It is more important for the law to protect the rights of religious people to live according to their faith and beliefs, even if that means some businesses won’t provide certain goods or services to gay and transgender people" (6).
* Asylum Prejudice Index (Exploratory):
* `t1_asyl_prej_therm`: 0-100 feeling thermometer: Someone who is seeking asylum in the U.S.
* `t1_asyl_prej_highcost`: Letting large numbers of asylum-seekers enter the U.S. comes at too high a cost for Americans.
* `t1_asyl_prej_contribute`: People who are seeking asylum in the U.S. will contribute to our country if given the chance.
* Asylum Policy Index (Exploratory):
* `t1_asyl_policy_asylum`: The U.S. should grant asylum to qualified individuals who are fleeing violence in their home countries.
* `t1_asyl_policy_asylumdeny`: The U.S. should require that any individual seeking asylum here has already applied for and been denied asylum in every country they’ve passed through to get here.
* Individual Exploratory Items Not In Index:
* `t1_explor_immprej_followrules`: Unauthorized immigrants should have followed the rules by coming the legal way. 
* `t1_explor_immprej_donewrong`: Anyone who is in the U.S. illegally did something wrong by coming here and/or staying here.
* `t1_action_imm_online`: These days people are busy and often don’t have time to do many of the things they would like to. Suppose in the next month someone asked you to engage in the following activities. How likely would you be to say yes?...Post in support of undocumented immigrants on Facebook, Twitter, or other social media. This is coded as a 5-point scale from "Not at all likely" (1) to "Extremely likely" (5).
* `t1_action_asyl_congress`: Same intro as above...Contact your Congressperson to call for a more orderly, humane asylum process. Coded same as above.
* `t1_action_lgbt_online`: Same intro as above...Post in support of gay and transgender people on Facebook, Twitter, or other social media. Coded same as above. This was not included in our pre-analysis plan although we use the same analytical strategy as the other outcome measures.
* `t1_action_lgbt_congress`: Same intro as above...Contact your Congressperson and ask them to support protecting gay and transgender people from discrimination at the workplace. Coded same as above. This was not included in our pre-analysis plan although we use the same analytical strategy as the other outcome measures.


We also asked three manipulation check outcomes (see below images for full wording). The first two were included in our pre-analysis plan as manipulation checks while we failed to mention the third outcome. However, we analyze this item using the same procedure as in our pre-analysis plan.

* `t1_recall_imm`: Checked "How the United States treats unauthorized immigrants and asylum seekers."
* `t1_recall_lgbt`: Checked "How businesses treat gay and lesbian people."
* `t1_knowledge_imm_taxes`: Checked "COMPLETELY TRUE" for "Undocumented immigrants pay state and local taxes".

![Wording of recall question](recall.png)

![Wording of knowledge of taxes question](taxes.png)

```{r eval = TRUE, include = FALSE}
# From PAP
t0.covariate <- c("t0_factor_immpolicy", "t0_factor_immprej",
                  "t0_factor_asyl_policy", "t0_factor_asyl_prej",
                  "t0_factor_lgbtpolicy", "t0_factor_lgbtprej",
                  "t0_therm_trans", "t0_therm_gay", "t0_therm_immigrant", "t0_therm_asylum",
                  "baseline_factor_general_politics",
                  "t0_economy_today", "t0_economy_yearfromnow", "t0_economy_personal",
                  "t0_knowledge_imm_taxes", "t0_college_educ", "t0_ideology", "t0_pid7",
                  "t0_trump_2020", "t0_congress_2020",
                  "t0_employ_fulltime", "t0_employ_parttime", "t0_live_city",
                  "vf_age", "vf_general18", "vf_general16", "vf_general14", "vf_general12",
                  "vf_dem", "vf_rep", "vf_female", "vf_white", "vf_afam", "vf_hispanic",
                  "state_CA", "state_CO", "state_MI", "state_NC", "state_TN")

# state_WI is excluded state category

t0.covariate.names <- c("Imm. Policy", "Imm. Prej.",
                        "Asyl. Policy", "Asyl. Prej.",
                        "LGBTQ Policy", "LGBTQ Prej.",
                        "Therm Transgender", "Therm Gay", "Therm Imm.", "Therm Asyl.",
                        "Politics Factor",
                        "Economy Today", "Economy Next Year", "Economy Personal",
                        "Imm. Pay Taxes", "College Educated", "Ideology", "PID-7",
                        "Trump Vote", "US House",
                        "Employed Full-time", "Employed Part-time", "Urbanicity",
                        "Age", "Voted '18", "Voted '16", "Voted '14", "Voted '12",
                        "Reg. Dem.", "Reg. Rep.", "Female", "White", "Af-Am", "Latino",
                        "CA", "CO", "MI", "NC", "TN")

x <- data[,c(t0.covariate)]
x <- as.matrix(x, dimnames = list(NULL, names(x)))
```

# Tests of Design Assumptions

## Covariate Balance

The below tables demonstrate that balance on pre-treatment observable attributes is maintained among the original universe of pre-survey respondents randomized to each group and the sub-sample that was successfully re-interviewed. Each table shows the mean value for the covariate under each condition as well as the *p*-value from a one-way ANOVA test with no multiple testing adjustment. All of these covariates were measured pre-treatment.

```{r eval=TRUE, echo=FALSE}

make.balance.table <- function(subset, varlist, names, caption){
  file <- subset(data, subset)
  balance <- matrix(ncol=3, nrow=length(varlist)+1)
  for(i in 1:length(varlist)){
    balance[i,1] <- mean(file[file$treat == 0,varlist[i]])
    balance[i,2] <- mean(file[file$treat == 1,varlist[i]])
    balance[i,3] <- mean(file[file$treat == 2,varlist[i]])
  }
  balance[length(varlist)+1,1] <- nrow(file[file$treat==0,])
  balance[length(varlist)+1,2] <- nrow(file[file$treat==1,])
  balance[length(varlist)+1,3] <- nrow(file[file$treat==2,])
  balance <- data.frame(round(balance, digits=2))
  rownames(balance) <- c(names, "N")
  anova.test.vector <- matrix(ncol=1, nrow=nrow(balance))
  for(i in 1:length(varlist)){
    anova.test.vector[i,1] <- round(summary(aov(file[,varlist[i]] ~ 
                                                  as.factor(treat), data = file))[[1]][["Pr(>F)"]][[1]], 2)
  }
  anova.test.vector[nrow(balance),1] <- "-"
  balance <- cbind(balance,anova.test.vector)
  colnames(balance) <- c("LGBTQ: Open to All", "Immigration: Eddie", "Immigration: MariCruz/Kandy", "p-value")
  return(kable(balance, longtable=T, caption = caption)  %>%
           kable_styling(latex_options = c("striped", "HOLD_position")))
} 
make.balance.table(data$t0_respondent == 1, t0.covariate, t0.covariate.names,
                   "Covariate Balance among Pre-Survey Respondents.")
```

```{r eval=TRUE, echo=FALSE}
make.balance.table(data$t1_respondent == 1, t0.covariate, t0.covariate.names,
                   "Covariate Balance among Post-Treatment Respondents.")
```
## Survey Attrition

Another design assumption is that the treatment does not affect the composition of the individuals who take each follow-up survey (Broockman, Kalla, and Sekhon 2017). We investigate this by regressing an indicator for responding to a post-treatment survey on indicators of treatment assignment. In the post-treatment survey, we find substantively small but borderline significant evidence of differential attrition. For this regression, the base condition is the LGBTQ ad.

```{r eval=TRUE, echo=FALSE}
t1.respondents <- summary(lm(t1_respondent ~ treat_eddie + treat_maricruz, data = data))$coef[2:3,]

overall.attrition <- data.frame(round(rbind(t1.respondents), 3))
names(overall.attrition) <- c("Effect", "SE", "t.stat", "p-value")
overall.attrition <- as.matrix(overall.attrition)
rownames(overall.attrition) <- rep(c("Immigration: Eddie", "Immigration: MariCruz/Kandy"), 1)


kable(overall.attrition, digits=2, caption = "Test for differential attrition") %>%
  kableExtra::group_rows("Post-Treatment Survey", 1, 2) %>%
  kable_styling(latex_options = c("striped", "HOLD_position"))
```

## Test of Differential Attrition by Covariates

```{r, eval=TRUE, echo=FALSE}
get.F.stat <- function(attrit.indicator){
  reduced.model <- lm(attrit.indicator ~ x + treat_eddie + treat_maricruz, data = data)
  xXt_eddie <- matrix(nrow = nrow(x), ncol = ncol(x))
  xXt_maricruz <- matrix(nrow = nrow(x), ncol = ncol(x))
  for(col in 1:ncol(x)) xXt_eddie[,col] <- as.numeric(data$treat_eddie) * x[,col]
  for(col in 1:ncol(x)) xXt_maricruz[,col] <- as.numeric(data$treat_maricruz) * x[,col]
  full.model <- lm(attrit.indicator ~ x + data$treat_eddie + xXt_eddie + data$treat_maricruz + xXt_maricruz)
  return(round(anova(reduced.model, full.model)[2,"Pr(>F)"], 2))
}

attrition <- matrix(, nrow = 1, ncol = 2)

attrition[1,2] <- get.F.stat(data$t1_respondent)

attrition[,1] <- c("Post-Treat Survey")
```
The above subsection demonstrated that there may have been some small average differential attrition; now, we test for whether the treatment caused attrition to differ by covariates (for example, whether it encouraged already-supportive subjects to complete the post-survey but also discouraged unsupportive subjects from doing so) (Gerber and Green 2012). To test whether attrition patterns are similar by covariates in treatment and placebo, we use a linear regression of whether or not an individual responded to the follow-up survey on treatment, baseline covariates, and treatment-covariate interactions. We then perform a heteroskedasticity-robust F-test of the hypothesis that all the interaction coefficients are zero. This procedure is standard practice (Gerber and Green 2012). 

The p-value on this F-test is `r attrition[1,2]`, suggesting there does not appear to be evidence of asymmetrical attrition.

# Results

Below we report the results in tabular form for each outcome measure. Each table includes two models: one in which we adjust for the pre-specified pre-treatment covariates to improve precision and a second unadjusted model. Note that we pre-registered a focus on the estimates with covariates (which were also pre-registered) since we expected these to be much more precise; the experimental design was intended to draw significant statistical power from the baseline survey. However, we also present results without covariates for completeness.

When we report the effects of the immigration ads, the LGBTQ ad is the comparison condition. When we report the effects of the LGBTQ ad, the immigration ad is the comparison condition.

## Effect of Immigration Ads

```{r, eval=TRUE, echo=FALSE}
# Function to compute clustered standard errors, from Mahmood Arai.
cl <- function(fm, cluster){
  M <- length(unique(cluster))
  N <- length(cluster)
  K <- fm$rank
  dfc <- (M/(M-1))*((N-1)/(N-K))
  uj  <- apply(estfun(fm), 2, function(x) tapply(x, cluster, sum))
  vcovCL <- dfc*sandwich(fm, meat=crossprod(uj)/N)
  coeftest(fm, vcovCL)
}

# Function to extract the ATE from OLS with clustered SEs.
est.ate.immigration <- function(dv, include.obs = NULL, include.covariates = TRUE){
  if(is.null(include.obs)){
    include.obs <- !is.na(dv) 
  }
  include.obs <- which(include.obs & !is.na(dv))
  
  if(include.covariates){
    lm.obj <- lm(dv[include.obs] ~ data$treat_eddie[include.obs] +
                   data$treat_maricruz[include.obs] +
                   x[include.obs,])
    # Calculate cluster-robust standard errors.
    result <- cl(lm.obj, data$hh_id[include.obs])[2:3,]
    result <- data.frame(result)
    names(result) <- c("Effect", "SE", "t-stat", "p-value")
    rownames(result) <- c("Eddie", "MariCruz/Kandy")
  }
  if(!include.covariates){
    lm.obj <- lm(dv[include.obs] ~ data$treat_eddie[include.obs] +
                   data$treat_maricruz[include.obs])
    # Calculate cluster-robust standard errors.
    result <- cl(lm.obj, data$hh_id[include.obs])[2:3,]
    result <- data.frame(result)
    names(result) <- c("Effect", "SE", "t-stat", "p-value")
    rownames(result) <- c("Eddie", "MariCruz/Kandy")
  }
  
  return(result)
}

est.ate.immigration.time <- function(dv, include.obs = NULL, include.covariates = TRUE){
  if(is.null(include.obs)){
    include.obs <- !is.na(dv) 
  }
  include.obs <- which(include.obs & !is.na(dv))
  
  if(include.covariates){
    lm.obj <- lm(dv[include.obs] ~ factor(data$survey_timing)[include.obs] + 
                   data$treat_eddie1[include.obs] +
                   data$treat_eddie2[include.obs] +
                   data$treat_eddie3[include.obs] +
                   data$treat_maricruz1[include.obs] +
                   data$treat_maricruz2[include.obs] +
                   data$treat_maricruz3[include.obs] +
                   x[include.obs,])
    # Calculate cluster-robust standard errors.
    result <- cl(lm.obj, data$hh_id[include.obs])[4:9,]
    result <- data.frame(result)
    names(result) <- c("Effect", "SE", "t-stat", "p-value")
    rownames(result) <- c("Eddie Timing 1", "Eddie Timing 2", "Eddie Timing 3",
                          "MariCruz/Kandy Timing 1", "MariCruz/Kandy Timing 2", "MariCruz/Kandy Timing 3")
  }
  if(!include.covariates){
    lm.obj <- lm(dv[include.obs] ~ factor(data$survey_timing)[include.obs] + 
                   data$treat_eddie1[include.obs] +
                   data$treat_eddie2[include.obs] +
                   data$treat_eddie3[include.obs] +
                   data$treat_maricruz1[include.obs] +
                   data$treat_maricruz2[include.obs] +
                   data$treat_maricruz3[include.obs])
    # Calculate cluster-robust standard errors.
    result <- cl(lm.obj, data$hh_id[include.obs])[4:9,]
    result <- data.frame(result)
    names(result) <- c("Effect", "SE", "t-stat", "p-value")
    rownames(result) <- c("Eddie Timing 1", "Eddie Timing 2", "Eddie Timing 3",
                          "MariCruz/Kandy Timing 1", "MariCruz/Kandy Timing 2", "MariCruz/Kandy Timing 3")
  }
  
  return(result)
}

make.results.table.ate.immigration <- function(dv, caption, subset = data$t1_respondent==1) {
  t1.ate.nocovars <- est.ate.immigration(data[,paste0("t1_",dv)], subset, include.covariates = FALSE)
  t1.ate.covars <- est.ate.immigration(data[,paste0("t1_",dv)], subset, include.covariates = TRUE)
  t1.ate.nocovars.time <- est.ate.immigration.time(data[,paste0("t1_",dv)], subset, include.covariates = FALSE)
  t1.ate.covars.time <- est.ate.immigration.time(data[,paste0("t1_",dv)], subset, include.covariates = TRUE)
  t1 <- round(cbind(t1.ate.covars, t1.ate.nocovars), 4)
  t1.time <- round(cbind(t1.ate.covars.time, t1.ate.nocovars.time), 4)
  t1 <- rbind(t1, t1.time)
  
  names(t1) <- rep(c("Effect", "SE", "t.stat", "p-value"), 2)
  t1 <- as.matrix(t1)
  return(kable(t1, digits=3, caption = caption) %>%
           add_header_above(c(" " = 1, "With Covariates" = 4, "Without Covariates" = 4)) %>%
           kableExtra::group_rows("Main Effects", 1, 2) %>%
           kableExtra::group_rows("By Survey Timing", 3, 8) %>%
           kable_styling(latex_options = c("striped", "HOLD_position")))
}
```


Below we estimate the ATE on ad recall. 

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration("recall_imm", "ATE effects on immigration ad recall")
```

Below we estimate the ATE on knowledge of taxes.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration("knowledge_imm_taxes", "ATE effects on knowledge of immigrants and taxes.")
```

Below we estimate the ATE on the overall index. 

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration("factor_immall", "ATE effects on immigration overall index")
```

Below we estimate the ATE on the overall prejudice index.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration("factor_immprej", "ATE effects on immigration prejudice index")
```

Below we estimate the ATE on the policy index.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration("factor_immpolicy", "ATE effects on immigration policy index")
```

Below we estimate the ATE on the immigration action-taking item.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration("action_imm_online", "ATE effects on immigration action-taking item.")
```

Below we estimate the ATE on the immigration action-taking item, limiting our analysis to those respondents who in the baseline survey were in the top tercile of support for undocumented immigrants. For example, among this subset, the average feeling thermometer rating towards undocumented immigrants was 67.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration("action_imm_online", "ATE effects on immigration action-taking item, top tercile.", data$t1_respondent == 1 & data$imm_top_tercile_baseline == 1)
```

Below we estimate the ATE on the asylum action-taking item.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration("action_asyl_congress", "ATE effects on asylum action-taking item.")
```

Below we estimate the ATE on the asylum action-taking item, limiting our analysis to those respondents who in the baseline survey were in the top tercile of support for undocumented immigrants.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration("action_asyl_congress", "ATE effects on asylum action-taking item, top tercile.", data$t1_respondent == 1 & data$imm_top_tercile_baseline == 1)
```


### Treatment Effect Heterogeneity by Respondent Partisanship

Below we report treatment effects by respondent partisanship, as measured in the initial baseline survey. We classify respondents as Democrats (including leaners), Republicans (including leaners), and Independents.

```{r, eval=TRUE, echo=FALSE}
data$t0_dem <- ifelse(data$t0_pid7 > 4 & data$t0_pid <= 7, 1, 0)
data$t0_rep <- ifelse(data$t0_pid7 >= 1 & data$t0_pid < 4, 1, 0)
data$t0_indep <- ifelse(data$t0_pid7 == 4, 1, 0)

make.results.table.ate.immigration.pid <- function(dv, caption) {
  t1.ate.dem <- est.ate.immigration(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_dem == 1, include.covariates = TRUE)
  t1.ate.rep <- est.ate.immigration(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_rep == 1, include.covariates = TRUE)
  t1.ate.indep <- est.ate.immigration(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_indep == 1, include.covariates = TRUE)
  
  t1 <- round(cbind(t1.ate.dem, t1.ate.rep, t1.ate.indep), 4)

  names(t1) <- rep(c("Effect", "SE", "t.stat", "p-value"), 3)
  t1 <- as.matrix(t1)
  
  return(kable(t1, digits=3, caption = caption) %>%
           add_header_above(c(" " = 1, "Democrats" = 4, "Republians" = 4, "Independents" = 4)) %>%
           kable_styling(latex_options = c("striped", "HOLD_position","scale_down")))
}


make.results.table.ate.immigration.pid("recall_imm", "HTE effects on immigration ad recall by respondent partisanship.")
make.results.table.ate.immigration.pid("knowledge_imm_taxes", "HTE effects on knowledge of immigrants and taxes by respondent partisanship.")
make.results.table.ate.immigration.pid("factor_immall", "HTE effects on immigration overall index by respondent partisanship.")
make.results.table.ate.immigration.pid("factor_immprej", "HTE effects on immigration prejudice index by respondent partisanship.")
make.results.table.ate.immigration.pid("factor_immpolicy", "HTE effects on immigration policy index by respondent partisanship.")
make.results.table.ate.immigration.pid("action_imm_online", "HTE effects on immigration action-taking item by respondent partisanship.")
make.results.table.ate.immigration.pid("action_asyl_congress", "HTE effects on asylum action-taking item by respondent partisanship.")
```


### Treatment Effect Heterogeneity by Respondent Political Knowledge

Below we report treatment effects by respondent political knowledge, as measured in the initial baseline survey. Specifically, we asked respondents six true/false questions:

* Undocumented immigrants pay state and federal taxes. (correct answer is true.)
* The federal minimum wage is $15 per hour. (correct answer is false.)
* The unemployment rate is 3.5%. (correct answer is true.)
* The United States is the world's largest economy. (correct answer is true.)
* More people are employed as yoga instructors than as coal miners. (correct answer is true.)
* The United States manufactures more cars than any other country. (correct answer is false.)

We then averaged respondents' answers to create a political knowledge scale. We then coded respondents as a 1 if they were above the median on this scale or a 0 if they were at or below the median. (This analysis was not pre-registered but was recommended by an anonymous reviewer.)

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.immigration.pk <- function(dv, caption) {
  t1.ate.above <- est.ate.immigration(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_pk_scale_above_median == 1, include.covariates = TRUE)
  t1.ate.below <- est.ate.immigration(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_pk_scale_above_median == 0, include.covariates = TRUE)

  t1 <- round(cbind(t1.ate.above, t1.ate.below), 3)

  names(t1) <- rep(c("Effect", "SE", "t.stat", "p-value"), 2)
  
  # Fix the rounding down of p-values to 0
  t1$`p-value`[t1$`p-value` == 0 ] <- "<0.001"
  names(t1) <- rep(c("Effect", "SE", "t.stat", "p-value"), 2)
  names(t1)[8] <- "p2"
  t1$`p2`[t1$`p2` == 0 ] <- "<0.001"
  names(t1)[8] <- "p-value"
  t1 <- as.matrix(t1)
  
  return(kable(t1, digits=3, caption = caption) %>%
           add_header_above(c(" " = 1, "Above Median" = 4, "At or Below Median" = 4)) %>%
           kable_styling(latex_options = c("striped", "HOLD_position")))
}


make.results.table.ate.immigration.pk("recall_imm", "HTE effects on immigration ad recall by respondent political knowledge")
make.results.table.ate.immigration.pk("knowledge_imm_taxes", "HTE effects on knowledge of immigrants and taxes by respondent political knowledge")
make.results.table.ate.immigration.pk("factor_immall", "HTE effects on immigration overall index by respondent political knowledge")
make.results.table.ate.immigration.pk("factor_immprej", "HTE effects on immigration prejudice index by respondent political knowledge")
make.results.table.ate.immigration.pk("factor_immpolicy", "HTE effects on immigration policy index by respondent political knowledge")
make.results.table.ate.immigration.pk("action_imm_online", "HTE effects on immigration action-taking item by respondent political knowledge")
make.results.table.ate.immigration.pk("action_asyl_congress", "HTE effects on asylum action-taking item by respondent political knowledge")
```



## Effect of LGBTQ Ad

```{r, eval=TRUE, echo=FALSE}
# Function to extract the ATE from OLS with clustered SEs.
est.ate.lgbt <- function(dv, include.obs = NULL, include.covariates = TRUE){
  if(is.null(include.obs)){
    include.obs <- !is.na(dv) 
  }
  include.obs <- which(include.obs & !is.na(dv))
  
  if(include.covariates){
    lm.obj <- lm(dv[include.obs] ~ data$treat_lgbt[include.obs] +
                   x[include.obs,])
    # Calculate cluster-robust standard errors.
    result <- cl(lm.obj, data$hh_id[include.obs])[2,]
    result <- data.frame(t(result))
    names(result) <- c("Effect", "SE", "t-stat", "p-value")
    rownames(result) <- c("LGBTQ")
  }
  if(!include.covariates){
    lm.obj <- lm(dv[include.obs] ~ data$treat_lgbt[include.obs])
    # Calculate cluster-robust standard errors.
    result <- cl(lm.obj, data$hh_id[include.obs])[2,]
    result <- data.frame(t(result))
    names(result) <- c("Effect", "SE", "t-stat", "p-value")
    rownames(result) <- c("LGBTQ")
  }
  
  return(result)
}

est.ate.lgbt.time <- function(dv, include.obs = NULL, include.covariates = TRUE){
  if(is.null(include.obs)){
    include.obs <- !is.na(dv) 
  }
  include.obs <- which(include.obs & !is.na(dv))
  
  if(include.covariates){
    lm.obj <- lm(dv[include.obs] ~ factor(data$survey_timing)[include.obs] + 
                   data$treat_lgbt1[include.obs] +
                   data$treat_lgbt2[include.obs] +
                   data$treat_lgbt3[include.obs] +
                   x[include.obs,])
    # Calculate cluster-robust standard errors.
    result <- cl(lm.obj, data$hh_id[include.obs])[4:6,]
    result <- data.frame(result)
    names(result) <- c("Effect", "SE", "t-stat", "p-value")
    rownames(result) <- c("LGBTQ Timing 1", "LGBTQ Timing 2", "LGBTQ Timing 3")
  }
  if(!include.covariates){
    lm.obj <- lm(dv[include.obs] ~ factor(data$survey_timing)[include.obs] + 
                   data$treat_lgbt1[include.obs] +
                   data$treat_lgbt2[include.obs] +
                   data$treat_lgbt3[include.obs])
    # Calculate cluster-robust standard errors.
    result <- cl(lm.obj, data$hh_id[include.obs])[4:6,]
    result <- data.frame(result)
    names(result) <- c("Effect", "SE", "t-stat", "p-value")
    rownames(result) <- c("LGBTQ Timing 1", "LGBTQ Timing 2", "LGBTQ Timing 3")
  }
  
  return(result)
}

make.results.table.ate.lgbt <- function(dv, caption, subset = data$t1_respondent==1) {
  t1.ate.nocovars <- est.ate.lgbt(data[,paste0("t1_",dv)], subset, include.covariates = FALSE)
  t1.ate.covars <- est.ate.lgbt(data[,paste0("t1_",dv)], subset, include.covariates = TRUE)
  t1.ate.nocovars.time <- est.ate.lgbt.time(data[,paste0("t1_",dv)], subset, include.covariates = FALSE)
  t1.ate.covars.time <- est.ate.lgbt.time(data[,paste0("t1_",dv)], subset, include.covariates = TRUE)
  t1 <- round(cbind(t1.ate.covars, t1.ate.nocovars), 4)
  t1.time <- round(cbind(t1.ate.covars.time, t1.ate.nocovars.time), 4)
  t1 <- rbind(t1, t1.time)

  # Fix the rounding down of p-values to 0
  t1$`p-value`[t1$`p-value` == 0 ] <- "<0.001"
  names(t1) <- rep(c("Effect", "SE", "t.stat", "p-value"), 2)
  names(t1)[8] <- "p2"
  t1$`p2`[t1$`p2` == 0 ] <- "<0.001"
  names(t1)[8] <- "p-value"
  t1 <- as.matrix(t1)
  
  return(kable(t1, digits=3, caption = caption) %>%
           add_header_above(c(" " = 1, "With Covariates" = 4, "Without Covariates" = 4)) %>%
           kableExtra::group_rows("Main Effects", 1, 1) %>%
           kableExtra::group_rows("By Survey Timing", 2, 4) %>%
           kable_styling(latex_options = c("striped", "HOLD_position")))
}
```

Below we estimate the ATE on ad recall. 

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.lgbt("recall_lgbt", "ATE effects on LGBTQ ad recall")
```

Below we estimate the ATE on the overall index. 

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.lgbt("factor_lgbtall", "ATE effects on LGBTQ overall index")
```

Below we estimate the ATE on the overall prejudice index.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.lgbt("factor_lgbtprej", "ATE effects on LGBTQ prejudice index")
```

Below we estimate the ATE on the policy index.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.lgbt("factor_lgbtpolicy", "ATE effects on LGBTQ policy index")
```

Below we estimate the ATE on the social media action-taking item.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.lgbt("action_lgbt_online", "ATE effects on LGBTQ social media action-taking item")
```

We also replicate this table, estimating the ATE on the social media action-taking item among those respondents who, at baseline, were in the top tercile of LGBTQ support. These respondents were supportive of LGBTQ rights at baseline. For example, their average feeling thermometer rating towards someone who is gay was 86.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.lgbt("action_lgbt_online", "ATE effects on LGBTQ social media action-taking item, top tercile", data$t1_respondent==1 & data$lgbt_top_tercile_baseline == 1)
```

Below we estimate the ATE on the Congress action-taking item.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.lgbt("action_lgbt_congress", "ATE effects on LGBTQ Congress action-taking item")
```

We also replicate this table, estimating the ATE on the Congress action-taking item among those respondents who, at baseline, were in the top tercile of LGBTQ support.

```{r, eval=TRUE, echo=FALSE}
make.results.table.ate.lgbt("action_lgbt_congress", "ATE effects on LGBTQ Congress action-taking item, top tercile", data$t1_respondent==1 & data$lgbt_top_tercile_baseline == 1)
```

Finally, we look at the effect of the LGBTQ ad on individual items while the ads were still running. This is an exploratory analysis. Note that all estimates are intent-to-treat effects on standardized outcome measures. We have coded all variables so that positive effects are in the pro-LGBTQ direction.

```{r, echo=FALSE, results = 'asis'}
get.itts <- function(dv) {
  ate.covars <- est.ate.lgbt(data[,paste0(dv)], data[,"t1_respondent"] == 1 & data[,"survey_timing"] == 1, include.covariates = TRUE)
  return(ate.covars)
}

policy.items <- c('t1_lgbtprej_trans_therm', 't1_lgbtprej_gay_therm',
                  't1_lgbtprej_overcorrect',
                  't1_lgbtprej_workclose', 't1_lgbtpolicy_discrim',
                  't1_lgbtpolicy_fire', 't1_lgbtpolicy_conflict',
                  't1_lgbtpolicy_refuse', 't1_lgbtpolicy_serveall',
                  't1_lgbtpolicy_moreimpt')

for (var in policy.items) {
  data[, paste0(var, '_std')] <- (data[,var] - mean(data[,var], na.rm = TRUE)) / sd(data[,var], na.rm = TRUE)
}

# These items are reverse coded
data$t1_lgbtprej_overcorrect_std <- -1*data$t1_lgbtprej_overcorrect_std
data$t1_lgbtpolicy_conflict_std <- -1*data$t1_lgbtpolicy_conflict_std
data$t1_lgbtpolicy_refuse_std <- -1*data$t1_lgbtpolicy_refuse_std
data$t1_lgbtpolicy_moreimpt_std <- -1*data$t1_lgbtpolicy_moreimpt_std

all.t1 <- rbind(get.itts('t1_lgbtprej_trans_therm_std'),
                get.itts('t1_lgbtprej_gay_therm_std'),
                get.itts('t1_lgbtprej_overcorrect_std'),
                get.itts('t1_lgbtprej_workclose_std'),
                get.itts('t1_lgbtpolicy_discrim_std'),
                get.itts('t1_lgbtpolicy_fire_std'),
                get.itts('t1_lgbtpolicy_conflict_std'),
                get.itts('t1_lgbtpolicy_refuse_std'),
                get.itts('t1_lgbtpolicy_serveall_std'),
                get.itts('t1_lgbtpolicy_moreimpt_std'))
  
names(all.t1) <- rep(c("Effect", "SE", "t.stat", "p-value"), 1)
all.t1 <- as.matrix(all.t1)
rownames(all.t1) <- policy.items

kable(all.t1, digits=3, caption = 'Effect on individual LGBTQ policy and prejudice items') %>%
  add_header_above(c(" " = 1, "With Covariates" = 4)) %>%
  kable_styling(latex_options = c("striped", "HOLD_position"))

```

### Treatment Effect Heterogeneity by Respondent Partisanship

Below we report treatment effects by respondent partisanship, as measured in the initial baseline survey. We classify respondents as Democrats (including leaners), Republicans (including leaners), and Independents.

```{r, eval=TRUE, echo=FALSE}


make.results.table.ate.lgbt.pid <- function(dv) {
  t1.ate.dem <- est.ate.lgbt(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_dem == 1, include.covariates = TRUE)
  t1.ate.rep <- est.ate.lgbt(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_rep == 1, include.covariates = TRUE)
  t1.ate.indep <- est.ate.lgbt(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_indep == 1, include.covariates = TRUE)
  
  t1 <- round(cbind(t1.ate.dem, t1.ate.rep, t1.ate.indep), 4)

  names(t1) <- rep(c("Effect", "SE", "t.stat", "p-value"), 3)
  t1 <- as.matrix(t1)
  return(t1)
}

pid.recall_lgbt <- make.results.table.ate.lgbt.pid("recall_lgbt")
pid.factor_lgbtall <- make.results.table.ate.lgbt.pid("factor_lgbtall")
pid.factor_lgbtprej <- make.results.table.ate.lgbt.pid("factor_lgbtprej")
pid.factor_lgbtpolicy <- make.results.table.ate.lgbt.pid("factor_lgbtpolicy")
pid.action_lgbt_online <- make.results.table.ate.lgbt.pid("action_lgbt_online")
pid.action_lgbt_congressl <- make.results.table.ate.lgbt.pid("action_lgbt_congress")


pid.results <- rbind(pid.recall_lgbt, pid.factor_lgbtall, pid.factor_lgbtprej, pid.factor_lgbtpolicy,
                     pid.action_lgbt_online, pid.action_lgbt_congressl)
    
kable(pid.results, digits=3, caption = "HTE effects on LGBTQ outcomes by respondent partisanship") %>%
           add_header_above(c(" " = 1, "Democrats" = 4, "Republians" = 4, "Independents" = 4)) %>%
           kableExtra::group_rows("Effect on LGBTQ ad recall", 1, 1) %>%
           kableExtra::group_rows("Effect on LGBTQ overall index", 2, 2) %>%
           kableExtra::group_rows("Effect on LGBTQ prejudice index", 3, 3) %>%
           kableExtra::group_rows("Effect on LGBTQ policy index", 4, 4) %>%
           kableExtra::group_rows("Effect on LGBTQ social media action-taking item", 5, 5) %>%
           kableExtra::group_rows("Effect on LGBTQ Congress action-taking item", 6, 6) %>%
           kable_styling(latex_options = c("striped", "HOLD_position","scale_down"))
```
### Treatment Effect Heterogeneity by Respondent Political Knowledge

Below we report treatment effects by respondent political knowledge. We classify respondents as above median or at or below median. Above we describe in detail how we measured political knowledge. (We did not pre-registered this analysis. It was recommended by an anonymous reviewer.)

```{r, eval=TRUE, echo=FALSE}


make.results.table.ate.lgbt.pk <- function(dv) {
  t1.ate.above <- est.ate.lgbt(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_pk_scale_above_median == 1, include.covariates = TRUE)
  t1.ate.below <- est.ate.lgbt(data[,paste0("t1_",dv)], data$t1_respondent==1 & data$t0_pk_scale_above_median == 0, include.covariates = TRUE)
  
  t1 <- round(cbind(t1.ate.above, t1.ate.below), 4)

  names(t1) <- rep(c("Effect", "SE", "t.stat", "p-value"), 2)
  t1 <- as.matrix(t1)
  return(t1)
}

pk.recall_lgbt <- make.results.table.ate.lgbt.pk("recall_lgbt")
pk.factor_lgbtall <- make.results.table.ate.lgbt.pk("factor_lgbtall")
pk.factor_lgbtprej <- make.results.table.ate.lgbt.pk("factor_lgbtprej")
pk.factor_lgbtpolicy <- make.results.table.ate.lgbt.pk("factor_lgbtpolicy")
pk.action_lgbt_online <- make.results.table.ate.lgbt.pk("action_lgbt_online")
pk.action_lgbt_congressl <- make.results.table.ate.lgbt.pk("action_lgbt_congress")


pk.results <- rbind(pk.recall_lgbt, pk.factor_lgbtall, pk.factor_lgbtprej, pk.factor_lgbtpolicy,
                     pk.action_lgbt_online, pk.action_lgbt_congressl)
    
kable(pk.results, digits=3, caption = "HTE effects on LGBTQ outcomes by respondent political knowledge") %>%
           add_header_above(c(" " = 1, "Above Median" = 4, "At or Below Median" = 4)) %>%
           kableExtra::group_rows("Effect on LGBTQ ad recall", 1, 1) %>%
           kableExtra::group_rows("Effect on LGBTQ overall index", 2, 2) %>%
           kableExtra::group_rows("Effect on LGBTQ prejudice index", 3, 3) %>%
           kableExtra::group_rows("Effect on LGBTQ policy index", 4, 4) %>%
           kableExtra::group_rows("Effect on LGBTQ social media action-taking item", 5, 5) %>%
           kableExtra::group_rows("Effect on LGBTQ Congress action-taking item", 6, 6) %>%
           kable_styling(latex_options = c("striped", "HOLD_position"))
```

## Multiple Testing

Our pre-analysis plan did not specify adjusting p-values for multiple testing. However, an anonymous reviewer suggested we assess whether the positive persuasive effects we observe of the LGBTQ ad while the advertisement was still running was a false positive. 

Because we did not pre-register a multiple testing procedure, we follow David McKenzie's advice^[https://blogs.worldbank.org/impactevaluations/overview-multiple-hypothesis-testing-commands-stata] for economists to control the Familywise Error Rate using the wyoung package in Stata (Jones et al. 2019), which implements the free step-down resampling methodology of Westfall and Young (1993), and to control the False Discovery Rate using Anderson (2008)'s Stata code for sharpened q-values, which implements the sharpened two-stage q-values from Benjamini, Krieger, and Yekutieli (2006). 

In calculating the sharpened q-values, we include all p-values on our main outcome indices, both from main effects and treatment-by-survey timing interactions. In total, this results in 68 p-values. There are six LGBTQ outcomes (recall, overall index, prejudice index, policy index, Congressional action-taking, and online action-taking), one main effect across all observations, and three subgroups for each survey wave. This results in 24 p-values of LGBTQ outcomes. Similarly, there are six immigration outcomes (recall, overall index, prejudice index, policy index, online action-taking, and tax knowledge), two treatments, one main effect across all observations, and three subgroups for each survey wave. We exclude any p-values (4) from the effect of the Eddie ad on tax knowledge because the Eddie ad did not mention this fact and therefore no effect was expected. This results in 44 p-values of immigration outcomes.

In calculating the Westfall and Young p-values, we limit our analysis to the family of p-values encompassing the effect of the LGBTQ ad while the ad was still airing. This results in six hypothesis tests: one for each of the LGBTQ outcomes listed above. We define this as a distinct family because we expected the effects of the ads to potentially be different depending on whether the ads were still airing or they had already come off the air (i.e., based on prior research, we expected to see decay).

The below table presents the adjusted p-values assessing the robustness of the finding of positive persuasive effects when the LGBTQ ad was still running to multiple testing adjustments. Overall, we find that the statistically significant results remain significant even after adjusting for multiple testing.

```{r table-simple, echo=FALSE, message=FALSE, warnings=FALSE, results='asis'}
# NOTE: These values are calculated in multiple testing adjustment for lgbt ad.do. 
# from https://stackoverflow.com/questions/19997242/simple-manual-rmarkdown-tables-that-look-good-in-html-pdf-and-docx
panderOptions('table.split.table', Inf)
set.caption("Multiple Testing Adjustment: LGBTQ Ad While Ad Still Airing")
my.data <- "| Outcome | Treatment Effect Coefficient | Covariate-adjusted p-value (no multiple testing correction) | Sharpened q-value | Westfall-Young adjusted p-value |
|---|---|---|---|---|
| Ad Recall | 0.063 | < 0.001 | 0.001 | <0.001 |
| Prejudice + Policy Index | 0.036 | 0.001 | 0.005 | 0.004 |
| Prejudice Index | 0.054 | < 0.001 | 0.002 | 0.002 |
| Policy Index | 0.023 | 0.064 | 0.183 | 0.168 |"
df <- read.delim(textConnection(my.data),header=FALSE,sep="|",strip.white=TRUE,stringsAsFactors=FALSE)
names(df) <- unname(as.list(df[1,])) # put headers on
df <- df[-1,] # remove first row
df <- df[,-1] # remove first column
df <- df[,-6] # remove last column
df <- df[-1,] # remove first row
row.names(df)<-NULL
pander(df, style = 'rmarkdown')
```


# Complier Average Causal Effects

Just because a household was randomly assigned to receive an ad does not mean that household watched the ad. For example, during the period of the experiment, that household may have never turned on their television. However, the advertising firm we partnered with on running the TV ads was able to collect data on how often a household saw an advertisement for 16,043 (51%) voters in the experiment. The remaining 49% have older television technologies that do not record this data, even if they view the advertisement. We exclude those voters with the older technology from the analyses presented in this section.

Among the 51% of voters with this technology, we do not know for certain whether someone saw the ad -- perhaps they left the TV to go to the bathroom -- but we at least know the television was on and the ad aired. We can therefore limit our analysis to these so-called compliers. Furthermore, the advertising firm also knows how many times each ad ran for each household. Note that this was not randomly assigned: however, we can measure observationally whether treatment effects vary with the number of exposures (dosage effects). This data is also only available for a subset of households. 

Due to data privacy reasons, we are unable to share the data on which households were compliers. We pre-wrote code for this section that the advertising firm then executed for us. 

The main descriptive findings on exposures are:

* LGBTQ Ad: 62% were exposed to the ad at least once. Of those exposed at least once, the median person was exposed 11 times (mean = 11.7; se = 0.13).
* Eddie Immigration Ad: 61% were exposed to the ad at least once. Of those exposed at least once, the median person was exposed 13 times (mean = 14.1; se = 0.17).
* MariCruz/Kandy Immigration Ad: 61% were exposed to the ad at least once. Of those exposed at least once, the median person was exposed 13 times (mean = 13.9; se = 0.17).



```{r imm graph no decay, include=FALSE, echo=FALSE, eval=TRUE}
## ITT
# Make immigration graph
imm.recall <- cbind(est.ate.immigration(data[,paste0("t1_", "recall_imm_std")], data$t1_respondent==1, include.covariates = TRUE), dv = "Recall")
imm.knowl <- cbind(est.ate.immigration(data[,paste0("t1_", "knowledge_imm_taxes_std")], data$t1_respondent==1, include.covariates = TRUE), dv = "Knowledge")
imm.overall <- cbind(est.ate.immigration(data[,paste0("t1_", "factor_immall")], data$t1_respondent==1, include.covariates = TRUE), dv = "Prej. + Policy Index")
imm.prej <- cbind(est.ate.immigration(data[,paste0("t1_", "factor_immprej")], data$t1_respondent==1, include.covariates = TRUE), dv = "Prejudice Index")
imm.policy <- cbind(est.ate.immigration(data[,paste0("t1_", "factor_immpolicy")], data$t1_respondent==1, include.covariates = TRUE), dv = "Policy Index")

imm.recall.time <- cbind(est.ate.immigration.time(data[,paste0("t1_", "recall_imm_std")], data$t1_respondent==1, include.covariates = TRUE), dv = "Recall")
imm.knowl.time <- cbind(est.ate.immigration.time(data[,paste0("t1_", "knowledge_imm_taxes_std")], data$t1_respondent==1, include.covariates = TRUE), dv = "Knowledge")
imm.overall.time <- cbind(est.ate.immigration.time(data[,paste0("t1_", "factor_immall")], data$t1_respondent==1, include.covariates = TRUE), dv = "Prej. + Policy Index")
imm.prej.time <- cbind(est.ate.immigration.time(data[,paste0("t1_", "factor_immprej")], data$t1_respondent==1, include.covariates = TRUE), dv = "Prejudice Index")
imm.policy.time <- cbind(est.ate.immigration.time(data[,paste0("t1_", "factor_immpolicy")], data$t1_respondent==1, include.covariates = TRUE), dv = "Policy Index")


## TOT
imm.results.tmp <- list(imm.recall, imm.knowl, imm.overall, imm.prej, imm.policy,
                    imm.recall.time, imm.knowl.time, imm.overall.time, imm.prej.time, imm.policy.time) %>% 
  map_df(rownames_to_column, 'Treatment') %>%
  separate(Treatment, c("Ad", "Timing"), " Timing ") %>%
  mutate(Timing = replace_na(Timing, "All")) %>%
  mutate(Timing = recode(Timing, `1` = "Ads Still Running", `2` = "Ads Off One Day", `3` = "Ads Off Three Days"))

imm_dv_name_map <- data.frame(
  DV = paste0('dv', c(31, 33, 34, 29, 35)),
  dv = c('Knowledge', 'Prejudice Index', 'Policy Index', 'Recall', 'Prej. + Policy Index')
)
ad_dv_name_map <- data.frame(
  Treat = c('treat_eddie', 'treat_maricruz'),
  Ad = c('Eddie + Prosperous Future', 'MariCruz/Kandy + Prosperous Future')
)

# Overall results
imm.cace.results <- read_csv('CACE/immigration_cace_results.csv')
imm.cace.results <- imm.cace.results %>%
  left_join(imm_dv_name_map) %>%
  left_join(ad_dv_name_map) %>%
  filter(dv != '' & Exposures == 'Any compliers') %>%
  dplyr::rename(Effect = Estimate) %>%
  mutate(Timing = 'All') %>%
  select(-Treat, -DV)

# Decay results
imm.decay.results <- readxl::read_xlsx('CACE/imm_cace_results_decay.xlsx')
est.and.se <- as.data.frame(str_split(str_trim(imm.decay.results$EstSE), ' ', simplify = T))
names(est.and.se) <- c('Effect', 'SE')
imm.decay.results <- imm.decay.results %>%
  cbind(est.and.se) %>%
  select(-EstSE) %>%
  mutate(Effect = as.numeric(Effect),
         SE = as.numeric(SE))

imm.decay.results <- imm.decay.results %>%
  mutate(Timing = '')

for(dv in unique(imm.decay.results$DV)) {
  for(ad in c('eddie', 'maricruz')) {
    # base effect = effect at time 1
    imm.decay.results$Timing[imm.decay.results$Treat == paste0('treat_', ad) & imm.decay.results$DV == dv] <- 'Ads Still Running'
    
    # 
    for(wave in 2:3) {
      var <- paste0('treat_', ad, ':survey_wave', wave)
      var_treat_only <- paste0('treat_', ad)
      base.treat.row <- which(imm.decay.results$Treat == var_treat_only & imm.decay.results$DV == dv)
      interaction.row <- which(imm.decay.results$Treat == var & imm.decay.results$DV == dv)
      
      est <- imm.decay.results$Effect[base.treat.row] +
        imm.decay.results$Effect[interaction.row]
      se <- sqrt(
        imm.decay.results$SE[interaction.row]^2 -
          imm.decay.results$SE[base.treat.row]^2
      )
      
      # Alter the DF
      imm.decay.results$Timing[interaction.row] <- paste0('Ads Off ',
                                                          ifelse(wave == 2, 'One Day', 'Three Days'))
      imm.decay.results$Effect[interaction.row] <- est
      imm.decay.results$SE[interaction.row] <- se
    }
  }
}

imm.decay.results <- imm.decay.results %>%
  mutate(Treat = str_replace(Treat, ':survey_wave2', '')) %>%
  mutate(Treat = str_replace(Treat, ':survey_wave3', '')) %>%
  mutate(Exposures = 'Any compliers') %>%
  left_join(imm_dv_name_map) %>%
  left_join(ad_dv_name_map) %>%
  select(-Treat, -DV)

imm.results <- rbind.fill(imm.cace.results, imm.results.tmp, imm.decay.results) %>%
  mutate(Exposures = case_when(
    Exposures == 'Any compliers' ~ 'Saw at least one ad (TOT, subjects with exposure data only)',
    TRUE ~ 'All respondents (ITT)'
  )) %>%
  select(-t.value, -p.value, -`t-stat`, -`p-value`)

imm.results <- imm.results %>% 
  mutate(Ad = recode(Ad, "Eddie" = "Eddie + Prosperous Future", "MariCruz/Kandy" = "MariCruz/Kandy + Prosperous Future"))

imm.results$Timing = factor(imm.results$Timing, ordered = T, levels = rev(c('Ads Off Three Days', 'Ads Off One Day', 'Ads Still Running',
                                                                            'All')))

imm.results$dv = factor(imm.results$dv, ordered = T, levels = rev(c('Policy Index', 'Prejudice Index', 'Prej. + Policy Index', 'Knowledge', 'Recall')))

# Rescale estimates and SEs to SDs
imm.results[imm.results$dv == 'Recall' & str_detect(imm.results$Exposures, 'TOT'), c('Effect', 'SE')] <- imm.results[imm.results$dv == 'Recall' & str_detect(imm.results$Exposures, 'TOT'), c('Effect', 'SE')] / .3173738 
imm.results[imm.results$dv == 'Knowledge' & str_detect(imm.results$Exposures, 'TOT'), c('Effect', 'SE')] <- imm.results[imm.results$dv == 'Knowledge' & str_detect(imm.results$Exposures, 'TOT'), c('Effect', 'SE')] /  .4880068 
```

Below we report treatment effects among those with exposure data who were exposed to the ad at least once (i.e., compliers).

```{r, echo=FALSE}
imm.results %>%
  filter(str_detect(Exposures, 'TOT')) %>%
  select(-Exposures) %>%
  mutate(Effect = round(Effect, digits = 4),
         SE = round(SE, digits = 4)) %>%
  arrange(dv, Timing, Ad) %>%
  kable(caption = "Effects of Immigration Ads Among Compliers (Those with Exposure Data Only)")  %>%
  kable_styling(latex_options = c("striped", "HOLD_position")) #"scale_down"
```

```{r imm graph, include=FALSE, echo=FALSE, eval=TRUE}
imm.graph <- imm.results %>%
  mutate(Exposures = str_wrap(Exposures, width = 20)) %>%
ggplot(aes(x=dv, y=Effect)) +
  geom_point(aes(shape=Timing, color = Timing, fill=Timing), position=position_dodge(width=0.5)) +
  geom_errorbar(width=.1, 
                aes(ymin = Effect - 1.96 * SE, ymax = Effect + 1.96 * SE, color = Timing), 
                position=position_dodge(width=0.5)) + 
  geom_errorbar(width=0, size = 1,
                aes(ymin = Effect - SE, ymax = Effect + SE, color = Timing), #, linetype = Timing 
                position=position_dodge(width=0.5)) + 
  xlab("") +
  ylab("Treatment Effect in Standard Deviations") +
  ggtitle("Effects of Immigration Ads") +
  geom_hline(yintercept=0, linetype = "dotted") + 
  theme_bw() +
  #coord_flip() +
  guides(colour = guide_legend(reverse=F), 
         fill = guide_legend(reverse=F),
         shape = guide_legend(reverse=F),
         linetype = guide_legend(reverse=F)) +
  theme(legend.title = element_blank(), legend.position="bottom", axis.text.x = element_text(angle = 45, vjust = .6)) +
  scale_x_discrete(labels = wrap_format(20)) +
  guides(fill = FALSE) +
  scale_fill_brewer(palette="RdGy") + #pick colors at https://www.r-graph-gallery.com/38-rcolorbrewers-palettes.html
  scale_color_brewer(palette="RdGy") + 
  facet_grid(Exposures ~ Ad, scales = 'free_y')

ggsave('imm_main_results_graph.pdf', imm.graph, scale = 1.35, height = 4, width = 6, units = 'in')
```


```{r lgbt graph, include=FALSE, echo=FALSE, eval=TRUE}
# ITT
# Make LGBTQ graph
lgbt.recall <- cbind(est.ate.lgbt(data[,paste0("t1_", "recall_lgbt_std")], data$t1_respondent==1, include.covariates = TRUE), dv = "Recall")
lgbt.overall <- cbind(est.ate.lgbt(data[,paste0("t1_", "factor_lgbtall")], data$t1_respondent==1, include.covariates = TRUE), dv = "Prej. + Policy Index")
lgbt.prej <- cbind(est.ate.lgbt(data[,paste0("t1_", "factor_lgbtprej")], data$t1_respondent==1, include.covariates = TRUE), dv = "Prejudice Index")
lgbt.policy <- cbind(est.ate.lgbt(data[,paste0("t1_", "factor_lgbtpolicy")], data$t1_respondent==1, include.covariates = TRUE), dv = "Policy Index")

lgbt.recall.time <- cbind(est.ate.lgbt.time(data[,paste0("t1_", "recall_lgbt_std")], data$t1_respondent==1, include.covariates = TRUE), dv = "Recall")
lgbt.overall.time <- cbind(est.ate.lgbt.time(data[,paste0("t1_", "factor_lgbtall")], data$t1_respondent==1, include.covariates = TRUE), dv = "Prej. + Policy Index")
lgbt.prej.time <- cbind(est.ate.lgbt.time(data[,paste0("t1_", "factor_lgbtprej")], data$t1_respondent==1, include.covariates = TRUE), dv = "Prejudice Index")
lgbt.policy.time <- cbind(est.ate.lgbt.time(data[,paste0("t1_", "factor_lgbtpolicy")], data$t1_respondent==1, include.covariates = TRUE), dv = "Policy Index")

lgbt.results <- list(lgbt.recall, lgbt.overall, lgbt.prej, lgbt.policy,
                     lgbt.recall.time, lgbt.overall.time, lgbt.prej.time, lgbt.policy.time) %>% 
  map_df(rownames_to_column, 'Treatment') %>%
  separate(Treatment, c("Ad", "Timing"), " Timing ") %>%
  mutate(Timing = replace_na(Timing, "All")) %>%
  mutate(Timing = recode(Timing, `1` = "Ads Still Running", `2` = "Ads Off One Day", `3` = "Ads Off Three Days"))

lgbt.results.tmp <- lgbt.results

## DECAY
lgbt_dv_name_map <- data.frame(
  DV = paste0('dv', c(30, 39, 40, 41)),
  dv = c('Recall', 'Prejudice Index', 'Policy Index', 'Prej. + Policy Index')
)

# Overall results
lgbt.cace.results <- read_csv('CACE/lgbt_cace_results.csv')
lgbt.cace.results <- lgbt.cace.results %>%
  left_join(lgbt_dv_name_map) %>%
  filter(dv != '' & Exposures == 'Any compliers') %>%
  dplyr::rename(Effect = Estimate) %>%
  mutate(Timing = 'All') %>%
  select(-Treat, -DV)

# Decay results
lgbt.decay.results <- read_csv('CACE/lgbt_cace_results_decay.csv') %>%
  mutate(Timing = '') %>%
  filter(!(Treat %in% paste0('survey_wave', 2:3))) %>%
  filter(Exposures == 'Any compliers') %>%
  filter(DV %in% lgbt_dv_name_map$DV)

for(dv in unique(lgbt.decay.results$DV)) {
    # base effect = effect at time 1
    lgbt.decay.results$Timing[lgbt.decay.results$Treat == 'treat_lgbt' & lgbt.decay.results$DV == dv] <- 'Ads Still Running'
    
    # 
    for(wave in 2:3) {
      var <- paste0('treat_lgbt:survey_wave', wave)
      var_treat_only <- 'treat_lgbt'
      base.treat.row <- which(lgbt.decay.results$Treat == var_treat_only & lgbt.decay.results$DV == dv)
      interaction.row <- which(lgbt.decay.results$Treat == var & lgbt.decay.results$DV == dv)
      
      est <- lgbt.decay.results$Estimate[base.treat.row] +
        lgbt.decay.results$Estimate[interaction.row]
      se <- sqrt(
        lgbt.decay.results$SE[interaction.row]^2 -
          lgbt.decay.results$SE[base.treat.row]^2
      )
      
      # Alter the DF
      lgbt.decay.results$Timing[interaction.row] <- paste0('Ads Off ',
                                                          ifelse(wave == 2, 'One Day', 'Three Days'))
      lgbt.decay.results$Estimate[interaction.row] <- est
      lgbt.decay.results$SE[interaction.row] <- se
    }
}


lgbt.decay.results <- lgbt.decay.results %>%
  mutate(Treat = str_replace(Treat, ':survey_wave2', '')) %>%
  mutate(Treat = str_replace(Treat, ':survey_wave3', '')) %>%
  mutate(Exposures = 'Any compliers') %>%
  left_join(lgbt_dv_name_map) %>%
  select(-Treat, -DV) %>%
  dplyr::rename(Effect = Estimate)

lgbt.results <- rbind.fill(lgbt.cace.results, lgbt.results, lgbt.decay.results) %>%
  mutate(Exposures = case_when(
    Exposures == 'Any compliers' ~ 'Saw at least one ad (TOT, subjects with exposure data only)',
    TRUE ~ 'All respondents (ITT)'
  ),
  Ad = 'LGBTQ') %>%
  select(-t.value, -p.value, -`t-stat`, -`p-value`)

lgbt.results$Timing = factor(lgbt.results$Timing, ordered = T, levels = rev(c('Ads Off Three Days', 'Ads Off One Day', 'Ads Still Running',
                                                                            'All')))

lgbt.results$dv = factor(lgbt.results$dv, ordered = T, levels = rev(c('Policy Index', 'Prejudice Index', 'Prej. + Policy Index', 'Recall')))

# rescale estimate and SE to SDs
lgbt.results[lgbt.results$dv == 'Recall' & str_detect(lgbt.results$Exposures, 'TOT'), c('Effect', 'SE')] <- lgbt.results[lgbt.results$dv == 'Recall' & str_detect(lgbt.results$Exposures, 'TOT'), c('Effect', 'SE')] / .2116426  
```


```{r, echo=FALSE}
lgbt.results %>%
  filter(str_detect(Exposures, 'TOT')) %>%
  select(-Exposures, -Ad) %>%
  mutate(Effect = round(Effect, digits = 4),
         SE = round(SE, digits = 4)) %>%
  arrange(dv, Timing) %>%
  kable(caption = "Effects of LGBTQ Ad Among Compliers (Those with Exposure Data Only)")  %>%
  kable_styling(latex_options = c("striped", "HOLD_position")) #"scale_down"
```

```{r, include=FALSE, echo=FALSE}
lgbt.results$Timing = factor(lgbt.results$Timing, ordered = T, levels = rev(c('Ads Off Three Days', 'Ads Off One Day', 'Ads Still Running',
                                                                              'All')))

lgbt.results$dv = factor(lgbt.results$dv, ordered = T, levels = rev(c('Policy Index', 'Prejudice Index', 'Prej. + Policy Index', 'Recall')))


lgbt.graph <- lgbt.results %>%
  mutate(Exposures = str_wrap(Exposures, width = 20)) %>%
  ggplot(aes(x=dv, y=Effect)) +
  geom_point(aes(shape=Timing, color = Timing, fill=Timing), position=position_dodge(width=0.5)) +
  geom_errorbar(width=.1, 
                aes(ymin = Effect - 1.96 * SE, ymax = Effect + 1.96 * SE, color = Timing), 
                position=position_dodge(width=0.5)) + 
  geom_errorbar(width=0, size = 1,
                aes(ymin = Effect - SE, ymax = Effect + SE, color = Timing), #, linetype = Timing 
                position=position_dodge(width=0.5)) + 
  xlab("") +
  ylab("Treatment Effect in Standard Deviations") +
  ggtitle("Effects of LGBTQ Ad") +
  geom_hline(yintercept=0, linetype = "dotted") + 
  theme_bw() +
  #coord_flip() +
  guides(colour = guide_legend(reverse=F), 
         fill = guide_legend(reverse=F),
         shape = guide_legend(reverse=F),
         linetype = guide_legend(reverse=F)) +
  theme(legend.title = element_blank(), legend.position="bottom") +
  scale_x_discrete(labels = wrap_format(20)) +
  guides(fill = FALSE) +
  scale_fill_brewer(palette="RdGy") + #pick colors at https://www.r-graph-gallery.com/38-rcolorbrewers-palettes.html
  scale_color_brewer(palette="RdGy") +
  facet_grid(Exposures ~ Ad, scales = 'free_y')

ggsave('lgbt_main_results_graph.pdf', lgbt.graph, scale = 1.35, height = 4, width = 6, units = 'in')
```

\clearpage

## Dosage Effects

Below we report dose-response relationships where we estimate separate treatment effects for the number of advertising exposures a household had, again only among the non-random subset of households where household-level records on the number of times the ad was shown to that household are available. Note that the number of exposures was not randomly assigned, so this dose-response analysis is observational, rather than causal (i.e., we can only estimate the effect for people who received different doses, but the number of doses itself is not randomly assigned).

The below figures show the dose response relationships. In each figure, the dotted red line is the intent-to-treat effect among all respondents. The solid blue line is the complier average causal effect among respondents who live in households that were exposed to the ad at least once. Each circle is the complier average causal effect for a particular number of ad exposures. The black line is a linear model across the number of exposures, weighted by the precision at each number of exposures.

### Immigration

The below figures examine the effects of the immigration ads. The figures reports that we find no increase in the ad's effectiveness among those respondents who live in households that were exposed to a greater number of advertisements. Using a linear model, on the overall index, we find a statistically insignificant decrease of -0.0009 standard deviations for each additional exposure to the Eddie and Prosperous Future advertisements (SE = 0.001, p = 0.48) and a similarly statistically insignificant increase of 0.0009 standard deviations for each additional exposure to the MariCruz/Kandy and Prosperous Future advertisements (SE = 0.001, p = 0.51). We also find no dosage effect on the other outcomes.

![Recall of Immigration Ad](CACE/fig_immigration/dv29 .pdf){width=50%}

![Knowledge that Immigrants Pay Taxes](CACE/fig_immigration/dv31 .pdf){width=50%}

![Effect on Immigration Prejudice Index](CACE/fig_immigration/dv33 .pdf){width=50%}

![Effect on Immigration Policy Index](CACE/fig_immigration/dv34 .pdf){width=50%}

![Effect on Immigration Overall Index](CACE/fig_immigration/dv35 .pdf){width=50%}

\clearpage

### LGBTQ

The below figures examine the effects of the LGBTQ ad. The figures reports that we find no increase in the ad's effectiveness among those respondents who live in households that were exposed to a greater number of advertisements. Using a linear model, we find a statistically insignificant increase of 0.0004 standard deviations on the overall index for each additional ad exposure (SE = 0.001, p = 0.77). We similarly find a lack of a dosage effect on the prejudice (d = 0.002, SE = 0.002, p = 0.29) and policy indexes (d = -0.0004, SE = 0.002, p = 0.80). However, on recall, we do find a meaningful dosage effect (d = 0.005, SE = 0.001, p < 0.001).

![Recall of LGBTQ Ad](CACE/fig_lgbt/dv30 .pdf){width=50%}

![Effect on LGBTQ Prejudice Index](CACE/fig_lgbt/dv39 .pdf){width=50%}

![Effect on LGBTQ Policy Index](CACE/fig_lgbt/dv40 .pdf){width=50%}

![Effect on LGBTQ Overall Index](CACE/fig_lgbt/dv41 .pdf){width=50%}

\clearpage

# References

* Anderson, Michael L. 2008. "Multiple Inference and Gender Differences in the Effects of Early Intervention: A Reevaluation of the Abecedarian, Perry Preschool, and Early Training Projects." Journal of the American Statistical Association 103(484): 1481-1495
* Benjamini, Yoav, Abba M. Krieger, and Daniel Yekutieli. 2006. "Adaptive Linear Step-up Procedures that Control the False Discovery Rate." Biometrika 93(3): 491-507
* Broockman, David E., Joshua L. Kalla, and Jasjeet S. Sekhon. 2017. "The design of field experiments with survey outcomes: A framework for selecting more efficient, robust, and ethical designs." Political Analysis 25(4): 435-464.
* Falk, Erika, Erin Grizard and Gordon McDonald. 2005. "Legislative issue advertising in the 108th Congress." The Annenberg Public Policy Center Report Series 47.
* Gerber, Alan S., James G. Gimpel, Donald P. Green, and Daron R. Shaw. 2011. "How large and long-lasting are the persuasive effects of televised campaign ads? Results from a randomized field experiment." American Political Science Review 105(1): 135-150.
* Gerber, Alan S., and Donald P. Green. 2012. Field experiments: Design, analysis, and interpretation. WW Norton.
* Jones, Damon, David Molitor, and Julian Reif. 2019 "What Do Workplace Wellness Programs Do? Evidence from the Illinois Workplace Wellness Study." Quarterly Journal of Economics 134(4): 1747-1791.
* Ridout, Travis N, Erika Franklin Fowler and Michael M Franz. 2021. "Spending Fast and Furious: Political Advertising in 2020." The Forum 18(4):465–492.
* Westfall, Peter H., and S. Stanley Young. 1993. Resampling-based Multiple Testing: Examples and Methods for p-value Adjustment. John Wiley & Sons, Inc.
* Zaller, John. 1996. "The Myth of Massive Media Impact Revived: New Support for a Discredited Idea." In Political Persuasion and Attitude Change , eds. Diana Mutz, Richard A. Brody, and Paul M. Sniderman. Ann Arbor: University of Michigan Press, 17-78.
